Skip to content

feat(evals): add full-workflow idempotency eval for triage-security re-runs#225

Merged
ruromero merged 1 commit into
RHEcosystemAppEng:mainfrom
ruromero:TC-4904
Jul 2, 2026
Merged

feat(evals): add full-workflow idempotency eval for triage-security re-runs#225
ruromero merged 1 commit into
RHEcosystemAppEng:mainfrom
ruromero:TC-4904

Conversation

@ruromero

@ruromero ruromero commented Jul 2, 2026 •
edited by sourcery-ai Bot
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • Adds eval ID 18 to evals/triage-security/evals.json — tests that re-running triage on an already-triaged CVE produces no duplicate mutations
  • Creates fixture vuln-issue-standard-already-triaged.md mirroring the standard CVE (TC-8001) with post-triage state: ai-cve-triaged label, In Progress status, existing remediation task links, digest comment, and summary comment
  • Five assertions cover idempotency of task creation, label addition, status transition, digest comments, and summary comments

Resolves TC-4904

Test plan

  • Eval 18 passes when run against the triage-security skill
  • Existing evals (1-17, 19-24) are unaffected
  • JSON is valid (verified via python3 -c "import json; json.load(open('evals/triage-security/evals.json'))")

🤖 Generated with Claude Code

Summary by Sourcery

Add a new triage-security evaluation to verify idempotent re-runs on already-triaged vulnerability issues.

New Features:

  • Introduce eval ID 18 for triage-security to ensure rerunning triage on an already-triaged CVE does not create duplicate artifacts.
  • Add an already-triaged vulnerability issue fixture capturing post-triage labels, status, links, and comments for idempotency testing.

@sourcery-ai

sourcery-ai Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Adds a new idempotency eval (ID 18) for triage-security by introducing a pre-triaged CVE fixture and wiring it into the triage-security evals JSON configuration.

File-Level Changes

Change Details Files
Add eval 18 definition to triage-security eval suite to verify idempotent re-runs on already-triaged CVEs.
  • Register new eval entry with ID 18 in evals.json for triage-security.
  • Configure eval 18 to use an already-triaged vulnerability issue fixture as input.
  • Document eval purpose as checking for absence of duplicate mutations across tasks, labels, status, digest comments, and summary comments.
 evals/triage-security/evals.json
Introduce synthetic already-triaged vulnerability issue fixture used by the new idempotency eval.
  • Create Markdown fixture modeling TC-8001 in post-triage state, including labels, status, custom fields, links, and comments.
  • Embed existing remediation task issue links to represent pre-existing upstream and downstream tasks.
  • Include prior digest and post-triage summary comments to ensure eval checks that re-runs do not duplicate them.
 evals/triage-security/files/vuln-issue-standard-already-triaged.md
Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

sourcery-ai[bot]
sourcery-ai Bot reviewed Jul 2, 2026
View reviewed changes

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've reviewed your changes and they look great!

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

github-actions[bot]
github-actions Bot reviewed Jul 2, 2026
View reviewed changes

@github-actions github-actions Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eval Results

Eval Results: triage-security

Eval Passed Failed Pass Rate
eval-1 11/11 0 100%
eval-10 5/5 0 100%
eval-11 5/5 0 100%
eval-12 5/5 0 100%
eval-13 5/5 0 100%
eval-14 5/5 0 100%
eval-15 5/5 0 100%
eval-16 7/7 0 100%
eval-17 5/5 0 100%
eval-18 5/5 0 100%
eval-19 5/5 0 100%
eval-2 5/5 0 100%
eval-20 4/4 0 100%
eval-21 4/4 0 100%
eval-22 4/4 0 100%
eval-23 4/4 0 100%
eval-24 4/4 0 100%
eval-25 4/4 0 100%
eval-3 5/5 0 100%
eval-4 5/5 0 100%
eval-5 6/6 0 100%
eval-6 6/6 0 100%
eval-7 5/5 0 100%
eval-8 8/8 0 100%
eval-9 5/5 0 100%

Pass rate: 100% · Tokens: 49,377 · Duration: 137s

Baseline (40c31ac8): 98% · 54,987 tokens · 132s

Generated by sdlc-workflow/run-evals v0.11.1

@ruromero

ruromero commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator Author

Verification Report for TC-4904 (commit 9442582)

Check Result Details
Review Feedback N/A No review comments to classify
Root-Cause Investigation N/A No sub-tasks created
Scope Containment PASS PR files match task spec exactly (2/2 files)
Diff Size PASS 94 lines across 2 files — proportionate for 1 eval entry + 1 fixture
Commit Traceability FAIL Commit 9442582 does not reference TC-4904 in headline or body
Sensitive Patterns PASS No secrets or credentials in added lines
CI Status PASS All 4 CI checks pass
Acceptance Criteria PASS 3/3 criteria met — eval 18 covers full re-run idempotency
Test Quality PASS Eval Quality: 100% (25/25 evals, up from 98% baseline)
Test Change Classification N/A No test files in diff
Verification Commands N/A No verification commands specified; no eval infrastructure changes

Overall: FAIL

Commit traceability failure: commit 9442582e uses headline feat(evals): add full-workflow idempotency eval for triage-security re-runs without referencing the Jira task ID TC-4904. The branch name provides implicit traceability but does not satisfy commit-level traceability.

All other checks pass. The new eval-18 passes 5/5 assertions at 100%, and the overall eval suite improved from 98% to 100% pass rate vs baseline.

This comment was AI-generated by sdlc-workflow/verify-pr v0.11.0.

@ruromero @claude
feat(evals): add full-workflow idempotency eval for triage-security r…
680a5c8 
…e-runs (TC-4904)

Add eval ID 18 with fixture vuln-issue-standard-already-triaged.md to
verify that running triage twice on the same CVE produces no duplicate
mutations — no new tasks, labels, links, or comments are created.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@ruromero

ruromero commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Verification Report for TC-4904 (commit 680a5c8)

Check Result Details
Review Feedback N/A No review comments to classify
Root-Cause Investigation N/A No sub-tasks created
Scope Containment PASS PR files match task spec exactly (2/2 files)
Diff Size PASS 94 lines across 2 files — proportionate for 1 eval entry + 1 fixture
Commit Traceability PASS Commit 680a5c8 references TC-4904 in headline
Sensitive Patterns PASS No secrets or credentials in added lines
CI Status PASS All 4 CI checks pass
Acceptance Criteria PASS 3/3 criteria met — eval 18 covers full re-run idempotency
Test Quality PASS Eval Quality: 100% (25/25 evals, up from 98% baseline)
Test Change Classification N/A No test files in diff
Verification Commands N/A No verification commands specified; no eval infrastructure changes

Overall: PASS

All checks pass. The new eval-18 passes 5/5 assertions at 100%, and the overall eval suite improved from 98% to 100% pass rate vs baseline.

Updated from prior FAIL report — commit traceability resolved by amending commit to include TC-4904 in the headline.

This comment was AI-generated by sdlc-workflow/verify-pr v0.11.0.

@ruromero ruromero merged commit be2d441 into RHEcosystemAppEng:main Jul 2, 2026
4 checks passed
@ruromero ruromero deleted the TC-4904 branch July 2, 2026 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments
@sourcery-ai sourcery-ai[bot] sourcery-ai[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ruromero